The widespread use of barcodes has greatly simplified supermarket checkout. However, many problems persist, causing both inconvenience for shoppers, and added costs for retailers.
One of the difficulties is finding a barcode on a package. While experienced supermarket clerks eventually learn barcode locations for popular products, even the best clerks sometimes have difficulty with less common products. For shoppers who use self-service checkout stations, any product can be confounding.
Another issue is that of re-orienting the package so that the barcode is in position for reading. Many items are straightforward. However, particularly with large items (e.g., a carton of diapers, or a heavy bag of dog food), it can be a physical challenge to manipulate the product so that the barcode is exposed to the reader. Often in self-service checkout stations, the physical constraints of the checkout station compound the difficulty, as these stations commonly don't have the handheld scanning capability with which conventional checkouts are equipped—forcing the shopper to manipulate the product so that barcode faces a glass scanning platen on the counter. (When properly positioned, the shopper may be unable to view either the platen or the barcode—exacerbating the difficulty.) Moreover, it is not enough for the barcode to be visible to the scanner; it must also be presented so as to roughly face the scanner (i.e., its surface normal must generally be within about 40-50 degrees of facing the scanning device in order to be read).
Sometimes a product is flipped and turned in search of a barcode, only to find there is none. Bottles of wine, for example, commonly lack barcodes.
Yet another issue is occasional difficulty in getting the scanning equipment to successfully read the barcode, after the barcode has been found and correctly positioned. This is a particular problem with malleable items (e.g., a package of frozen peas), in which the barcoded surface is crinkled or otherwise physically distorted.
To redress such issues, some have proposed identifying products with passive tags that can be sensed by radio (e.g., RFID and NFC chips). However, the costs of these tags are an obstacle in the low-margin grocery business. And it can be difficult to distinguish the responses from several different items on a checkout counter. Moreover, certain materials in the check-out queue may be radio-opaque—preventing some identifiers from being read. Privacy issues raise yet further concerns.
Other checkout technologies have also been tried. For example, in patent publication 20040081799, Kodak describes how a marking can be applied to supermarket packaging by adding a polymer layer that defines scannable information in the form of matte and glossy areas. The matte/glossy areas can form indicia such as barcodes, or digital watermarks. However, this technology requires applying a polymer layer to the packaging—a further expense, and an additional processing step that packagers are not presently equipped to provide.
Other identification technologies have been proposed for use in conjunction with barcode-based product identification. For example, patent application 20040199427 proposes capturing 2D imagery of products, and checking their color histograms against histograms associated with products identified by sensed barcode data, to ensure correct product identification. The same publication similarly proposes weighing articles on the conveyor—again checking for consistency with the barcode-indicated product. Publications 20040223663 and 20090060259 teach related arrangements, in which imagery of products is used to check for possibly switched barcodes.
U.S. Pat. No. 7,044,395 teaches that a watermark can replace a barcode, such as a UPC symbol or other standard product code, in a retail point of sale application. A reader unit at a checkout counter extracts a product identifier from the watermark, and uses it to look up the product and its price.
U.S. Pat. No. 4,654,872 describes a system employing two video cameras, which captures images of a 3D article, and uses the imagery to recognize the article. U.S. Pat. No. 7,398,927 teaches another two-camera system, this one to read product codes from articles despite specular reflections. U.S. Pat. No. 7,909,248 details a self-service checkout terminal in which captured imagery is compared against a database of reference imagery to try to identify a matching product.
In accordance with various embodiments of the present technology, certain drawbacks of the prior art are overcome, and new capabilities are provided.
For example, in one aspect, the present technology involves marking product packaging with a digital watermark that encodes related information (e.g., Universal Product Codes, such as UPC-A or UPC-E; Electronic Product Codes—EPC, European Article Number Codes—EAN, a URI or web address, etc.). The marking spans a substantial part of the packaging surface area, so that it can be sensed from one or more fixed cameras at a checkout station without repositioning of the item. The watermark indicia can be applied to the packaging along with other printing—integrated in the other packaging artwork.
In one such embodiment, a variety of recognition technologies are used at a checkout station—looking for different indicia of product identification (watermark, barcode, color histogram, weight, temperature, etc.). The system applies a set of rules to the collected evidence, and outputs a product identification based on the available information.
In another aspect, crinkles and other deformations in malleable product packaging are optically sensed, and are used in decoding an identifier from the distorted surface (e.g., the crinkled surface can be virtually flattened prior to decoding the identifier). In one particular such arrangement, the crinkled configuration is sensed by structure-from-motion techniques. In another, the product configuration is sensed by a structured light scanner (e.g., of the sort popularized by the Microsoft Kinect sensor).
In yet another aspect, a checkout station comprises a conveyor belt that includes markings that are optically sensed, and which are used to increase check-out speed and accuracy.
In still another aspect, imagery captured from an item being conveyor-transported at a checkout station is processed to compensate for motion blur, prior to applying a product recognition technology.
In yet another aspect, a plenoptic camera system senses information at a checkout station. The collected light field data is then processed to yield multiple different planes of focused imagery, to which product recognition technologies are applied. In some embodiments, these planes include a variety of non-parallel planes.
In still another aspect, 2D imagery acquired at a checkout station is applied to a GPU, which computes multiple perspective-transformed versions of the imagery—which are then analyzed for product recognition purposes. The GPU can process input imagery of several different focal lengths, e.g., captured by plural fixed-focus cameras, or by a camera that cyclically changes its focal plane, or by plenoptic sensing.
The foregoing and other features and advantages of the present technology will be more readily apparent from the following detailed description, which proceeds with reference to the accompanying drawings.